Imaging system and method for producing images using means for adjusting optical focus

ABSTRACT

An imaging system for producing images for a display apparatus. Imaging system includes at least one imaging unit including camera, optical element including first optical portion and second optical portion having different focal lengths, and means for adjusting optical focus; means for generating depth or voxel map; and processor. Processor is configured to obtain gaze direction of user; determine optical depth of object present in region of interest within real-world scene; and control means for adjusting optical focus of imaging unit, based on optical depth of object and focal lengths of first and second optical portions, to capture warped image of real-world scene, the warped image having spatially-uniform angular resolution.

TECHNICAL FIELD

The present disclosure relates generally to imaging systems; and morespecifically, to imaging systems for producing images for displayapparatuses. Moreover, the present disclosure relates to methods forproducing images for display apparatuses via aforesaid imaging systems.

BACKGROUND

Presently, several technologies (for example, such as virtual reality(VR), augmented reality (AR), mixed reality (MR) and extended reality(XR)) are being used to present interactive simulated environments tousers. The users utilize specialized Head-Mounted Devices (HMDs) forexperiencing and interacting with such simulated environments.Conventional HMDs display images that collectively constitute suchsimulated environments, to the user. When such images are capturedaccording to a gaze direction of the user, the simulated environmentswould appear realistic to the user.

In order to capture gaze-contingent images, various types of imagingequipment and techniques are currently being employed. Generally,imaging equipment and techniques employ optical components (such aslenses, mirrors, and the like) having uniform optical properties.Nowadays, specialized optical components having variable opticalproperties with respect to magnification and/or de-magnification arebeing developed for use in such imaging equipment and techniques.Notably, these specialized optical components capture warped images ofthe given environment by magnifying a first portion of the givenenvironment to a greater degree than a second portion of the givenenvironment. Often, such imaging equipment and techniques employautofocusing mechanism to adjust optical focus of said imaging equipmentbased on the gaze direction of the user.

However, such imaging equipment and techniques employing specializedoptical components and autofocusing mechanism have certain limitationsassociated therewith. The autofocusing mechanism of such imagingequipment and techniques have a low autofocus speed based on the gazedirection of the user. As a result, a large amount of time is requiredfor autofocusing based on the gaze direction of the user. Therefore,generation of the warped images using such imaging equipment andtechniques is very time consuming. In such a case, when the autofocusspeed of the autofocusing mechanism is increased using conventionaltechniques, the generated warped images appear blurred. As a result, thegenerated warped images using such imaging equipment and techniques aresuboptimal.

Therefore, in light of the foregoing discussion, there exists a need toovercome the aforementioned drawbacks associated with generating imagesfor display apparatuses.

SUMMARY

The present disclosure seeks to provide an imaging system for producingimages for a display apparatus. The present disclosure also seeks toprovide a method for producing images for a display apparatus. Thepresent disclosure seeks to provide a solution to the existing problemassociated with a low autofocus speed of autofocusing mechanismsemployed by conventional imaging equipment and techniques. An aim of thepresent disclosure is to provide a solution that overcomes at leastpartially the problems encountered in prior art, and provides anefficient imaging system that produces images for the display apparatususing an extremely high autofocus speed.

In one aspect, an embodiment of the present disclosure provides animaging system for producing images for a display apparatus, the imagingsystem comprising:

at least one imaging unit, a given imaging unit comprising:

-   -   a camera;    -   an optical element that comprises at least a first optical        portion and a second optical portion having different focal        lengths; and    -   means for adjusting an optical focus of the given imaging unit;

means for generating a depth or voxel map of a given real-world scene;and

a processor communicably coupled to the at least one imaging unit andsaid means for generating, wherein the processor is configured to:

-   -   obtain, from the display apparatus, information indicative of a        gaze direction of a user;    -   determine, based on the gaze direction of the user and the depth        or voxel map of the given real-world scene, an optical depth of        at least one object present in a region of interest within the        given real-world scene; and    -   control the means for adjusting the optical focus of the given        imaging unit, based on the optical depth of the at least one        object and the focal lengths of the first optical portion and        the second optical portion, to capture at least one warped image        of the given real-world scene, the at least one warped image        having a spatially-uniform angular resolution.

In another aspect, an embodiment of the present disclosure provides amethod for producing images for a display apparatus, the method beingimplemented via an imaging system comprising at least one imaging unit,a given imaging unit comprising a camera, an optical element thatcomprises at least a first optical portion and a second optical portionhaving different focal lengths, and means for adjusting an optical focusof the given imaging unit, the method comprising:

-   -   obtaining, from the display apparatus, information indicative of        a gaze direction of a user;    -   generating a depth or voxel map of a given real-world scene;    -   determining, based on the gaze direction of the user and the        depth or voxel map of the given real-world scene, an optical        depth of at least one object present in a region of interest        within the given real-world scene; and    -   adjusting an optical focus of the given imaging unit, based on        the optical depth of the at least one object and the focal        lengths of the first optical portion and the second optical        portion, to capture at least one warped image of the given        real-world scene, the at least one warped image having a        spatially-uniform angular resolution.

Embodiments of the present disclosure substantially eliminate or atleast partially address the aforementioned problems in the prior art,and enables high speed adjustment of optical focus within the imagingsystem for producing gaze-contingent warped images in real time ornear-real time for a display apparatus.

Additional aspects, advantages, features and objects of the presentdisclosure would be made apparent from the drawings and the detaileddescription of the illustrative embodiments construed in conjunctionwith the appended claims that follow.

It will be appreciated that features of the present disclosure aresusceptible to being combined in various combinations without departingfrom the scope of the present disclosure as defined by the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The summary above, as well as the following detailed description ofillustrative embodiments, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating the presentdisclosure, exemplary constructions of the disclosure are shown in thedrawings. However, the present disclosure is not limited to specificmethods and instrumentalities disclosed herein. Moreover, those skilledin the art will understand that the drawings are not to scale. Whereverpossible, like elements have been indicated by identical numbers.

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the following diagrams wherein:

FIGS. 1, 2 and 3 illustrate block diagrams of architectures of animaging system for producing images for a display apparatus, inaccordance with various embodiments of the present disclosure;

FIG. 4 illustrates an exemplary real-world scene including two objectsat different optical depths, said two objects being captured using asingle imaging unit, in accordance with an embodiment of the presentdisclosure;

FIG. 5 illustrates an exemplary real-world scene including two objectsat different optical depths, said two objects being captured using twoimaging units, in accordance with an embodiment of the presentdisclosure;

FIG. 6 is an example graphical representation of depth of field of agiven imaging unit as a function of focal length of an optical elementof the given imaging unit, in accordance with an embodiment of thepresent disclosure;

FIG. 7 is an example graphical representation of how focus is adjustedvia an imaging system, in accordance with an embodiment of the presentdisclosure;

FIG. 8 is an example implementation of a display apparatus, inaccordance with an embodiment of the present disclosure;

FIG. 9 is an example implementation of a given imaging unit, inaccordance with an embodiment of the present disclosure;

FIG. 10 is a schematic illustration of an example implementation where asymmetrical optical element is rotated with respect to a camera, inaccordance with an embodiment of the present disclosure;

FIG. 11 is a schematic illustration of another example implementationwhere an asymmetrical optical element is rotated with respect to acamera, in accordance with another embodiment of the present disclosure;and

FIG. 12 illustrates steps of a method for producing images for a displayapparatus, in accordance with an embodiment of the present disclosure.

In the accompanying drawings, an underlined number is employed torepresent an item over which the underlined number is positioned or anitem to which the underlined number is adjacent. A non-underlined numberrelates to an item identified by a line linking the non-underlinednumber to the item. When a number is non-underlined and accompanied byan associated arrow, the non-underlined number is used to identify ageneral item at which the arrow is pointing.

DETAILED DESCRIPTION OF EMBODIMENTS

The following detailed description illustrates embodiments of thepresent disclosure and ways in which they can be implemented. Althoughsome modes of carrying out the present disclosure have been disclosed,those skilled in the art would recognize that other embodiments forcarrying out or practising the present disclosure are also possible.

In one aspect, an embodiment of the present disclosure provides animaging system for producing images for a display apparatus, the imagingsystem comprising:

at least one imaging unit, a given imaging unit comprising:

-   -   a camera;    -   an optical element that comprises at least a first optical        portion and a second optical portion having different focal        lengths; and    -   means for adjusting an optical focus of the given imaging unit;

means for generating a depth or voxel map of a given real-world scene;and

a processor communicably coupled to the at least one imaging unit andsaid means for generating, wherein the processor is configured to:

-   -   obtain, from the display apparatus, information indicative of a        gaze direction of a user;    -   determine, based on the gaze direction of the user and the depth        or voxel map of the given real-world scene, an optical depth of        at least one object present in a region of interest within the        given real-world scene; and    -   control the means for adjusting the optical focus of the given        imaging unit, based on the optical depth of the at least one        object and the focal lengths of the first optical portion and        the second optical portion, to capture at least one warped image        of the given real-world scene, the at least one warped image        having a spatially-uniform angular resolution.

In another aspect, an embodiment of the present disclosure provides amethod for producing images for a display apparatus, the method beingimplemented via an imaging system comprising at least one imaging unit,a given imaging unit comprising a camera, an optical element thatcomprises at least a first optical portion and a second optical portionhaving different focal lengths, and means for adjusting an optical focusof the given imaging unit, the method comprising:

-   -   obtaining, from the display apparatus, information indicative of        a gaze direction of a user;    -   generating a depth or voxel map of a given real-world scene;    -   determining, based on the gaze direction of the user and the        depth or voxel map of the given real-world scene, an optical        depth of at least one object present in a region of interest        within the given real-world scene; and    -   adjusting an optical focus of the given imaging unit, based on        the optical depth of the at least one object and the focal        lengths of the first optical portion and the second optical        portion, to capture at least one warped image of the given        real-world scene, the at least one warped image having a        spatially-uniform angular resolution.

The present disclosure provides the aforementioned imaging system andthe aforementioned method for producing images for the displayapparatus. The images produced by the imaging system are gaze-contingentsince the imaging system efficiently utilizes the information indicativeof the gaze direction of the user for adjusting optical focus of thegiven imaging unit. Therefore, when the user is presented the producedimages by the display apparatus, the user experiences considerableimmersion within given real-world scene. The imaging system implementsboth physical adjustments and processing-based adjustments to providethe user with a high autofocus speed based on the detected gazedirection of the user, thereby improving the users experience of asimulated environment. Beneficially said adjustments are made in realtime or near-real time, and therefore the at least one warped image isgenerated at a very high speed. Moreover, the method described herein iscomputationally efficient.

Throughout the present disclosure, the term “imaging system” refers tospecialized equipment for producing images for the display apparatus. Itwill be appreciated that the imaging system produces said images in realtime or near real time.

Throughout the present disclosure, the term “display apparatus” refersto specialized equipment that is configured to present a simulatedenvironment to the user when the display apparatus in operation is wornby the user on his/her head. In such an instance, the display apparatusacts as a device (for example, such as a virtual reality headset, a pairof virtual reality glasses, an augmented reality headset, a pair ofaugmented reality glasses, a mixed reality headset, a pair of mixedreality glasses, an extended reality headset, a pair of extended realityglasses and so forth) that is operable to present a visual scene of thesimulated environment to the user. The display apparatus may alsocommonly be referred to as “head-mounted display apparatus”.

The imaging system is at least communicably coupled to the displayapparatus. By way of such communicable coupling, the imaging systemtransmits the produced images to the display apparatus. In someimplementations, the imaging system is integrated with the displayapparatus. In such implementations, the imaging system is physicallycoupled to the display apparatus (for example, attached via mechanicaland electrical connections to components of the display apparatus). Inother implementations, the imaging system is implemented on a remotedevice that is separate from the display apparatus. In suchimplementations, the imaging system and the display apparatus arecommunicably coupled via a wired communication interface or a wirelesscommunication interface. Optionally, the imaging system is mounted onthe remote device. Examples of the remote device include, but are notlimited to, a drone, a vehicle, and a robot. In such an instance, theremote device is physically positioned at a given real-worldenvironment, whereas the user of the display apparatus is positionedaway from (for example, at a distance from) the remote device.

Throughout the present disclosure, the term “imaging unit” refers toequipment configured to capture the at least one warped image of thegiven real-world scene, wherein the at least one warped image is to beutilized by the display apparatus. It will be appreciated that the term“at least one imaging unit” refers to “one imaging unit” in someimplementations, and “a plurality of imaging units” in otherimplementations.

Throughout the present disclosure, the term “camera” refers to equipmentthat is operable to detect and process light reflected from the givenreal-world scene, so as to capture the at least one warped image of thegiven real-world scene. Optionally, the camera comprises a camera chip,wherein the light from the given real-world scene is directed by theoptical element onto a photosensitive surface of the camera chip,thereby enabling the camera to capture the at least one warped image ofthe given real-world scene. Optionally, the camera is implemented as atleast one of: a Red-Green-Blue (RGB) camera, a RGB-Depth (RGB-D) camera,a Red-Green-Infrared-Blue (RGIRB) camera, a Red-Green-Complementary-Blue(RGYB) camera, a stereo camera, a plenoptic camera. In the RGYB camera,the complementary ‘Y’ color is either a complementary color of greencolor or a wide wavelength green colour.

Throughout the present disclosure, the term “optical element” refers toa configuration of one or more optical components (for example, such aslenses, prisms, mirrors and so forth) that is capable of modifying thelight passing therethrough or reflecting therefrom. The optical elementis arranged on an optical path of light emanating from the givenreal-world scene, between the given real-world scene and the camera.Optionally, the optical element is arranged in a manner that the lightfrom the given real-world scene is directed onto the photosensitivesurface of the camera chip of the camera, thereby enabling the camera tocapture the at least one warped image of the given real-world scene.

The terms “first optical portion” and “second optical portion” refer todifferent portions of the optical element having different focallengths. Moreover, the term “focal length” refers to an optical distancebetween a point within the optical element where light rays emanatingfrom the given real-world scene converge and the camera (morespecifically, the photosensitive surface of the camera chip) of the atleast one imaging unit, when the optical element is focused at infinity.A projection of a first region of the given real-world scene passesthrough or reflects from the first optical portion, while a projectionof a second region of the given real-world scene passes through orreflects from the second optical portion, when the at least one imagingunit captures the at least one warped image. The projections of thefirst region and the second region of the given-real world scenecorrespond to a first warped portion and a second warped portion of theat least one warped image, respectively.

Optionally, the optical element is implemented as at least one of: alens, a mirror, a prism. Optionally, the optical element is implementedas a single lens having a complex shape. As an example, such a lens mayhave an aspheric shape. Optionally, the single lens is implemented asany of: a Fresnel lens, a Liquid Crystal (LC) lens or a liquid lens.Alternatively, optionally, the optical element is implemented as asingle mirror having a complex shape. As an example, a reflectivesurface of such a mirror may have an aspheric shape. Yet alternatively,optionally, the optical element is implemented as a configuration ofmultiple lenses and/or mirrors. Optionally, in such a case, the firstoptical portion and the second optical portion are implemented asseparate optical elements.

Optionally, the optical element is asymmetrical with respect to itsoptical axis. In such a case, the first optical portion and the secondoptical portion are positioned asymmetrically with respect to theoptical axis of the optical element. Alternatively, optionally, theoptical element is symmetrical with respect to its optical axis. In sucha case, the second optical portion surrounds an optical center of theoptical element, wherein a center of the second optical portioncoincides with the optical center of the optical element. Moreover, thefirst optical portion surrounds the second optical portion, and thefirst optical portion is surrounded by a periphery of the opticalelement. Optionally, when the optical element is symmetrical withrespect to its optical axis, the first optical portion and the secondoptical portion are concentric to each other.

It will be appreciated that the first and second optical portions of theoptical element have different focal lengths. In other words, theoptical element has a variable focal length. Notably, the focal lengthof a given optical portion is inversely related to an optical power ofthe given optical portion.

It will be appreciated that since the first and second optical portionsof the optical element have different focal lengths, the first andsecond optical portions have different focal planes (namely, planes atwhich said optical elements focus) within the given real-world scene.Moreover, the first and second optical portions of the optical elementhave different depths of field. Notably, the depth of field of a givenoptical portion is inversely proportional to a focal length of the givenoptical portion. The depths of field of the first and second opticalportions change differently as a function of movement of the opticalelement due to the variable focal length of the optical element. As anexample, a magnitude of change in optical focus (for example, incentimeters) within the given real-world scene when the means foradjusting focus moves the optical element by a specific distance (forexample, in micrometers) is different for different optical portions ofthe optical element. Consequently, the depth of field varies as well, asa function of gaze direction (since the optical focus is adjusted basedon the detected gaze direction).

Optionally, the focal length of the second optical portion is greaterthan the focal length of the first optical portion. Optionally, in thisregard, the focal length of the optical element varies as a function ofangular distance from the center of the second optical portion.Optionally, in this regard, the focal length is maximum along an axispassing through the center of the second optical portion, and decreaseson going away from said center. It will be appreciated that since ageometry of the optical element is constant, its focal length in eachdirection is pre-known (notably, by measurement or calculation). In anexample, the focal length of the optical element along the axis passingthrough the center of the second optical portion may be 17 millimeters,whereas the focal length of the optical element at an angular distanceof 30 degrees from said axis may be 5 millimeters. One such examplevariation of the focal length of the optical element has beenillustrated in conjunction with FIG. 6, as described below.

Furthermore, a focal length of a given optical portion determines anextent to which a projection of a given region of the given real-worldscene would appear magnified when viewed through the given opticalportion. The first and second optical portions have different opticalproperties with respect to magnification. Notably, the first and secondoptical portions are capable of differently magnifying the projectionsof the first and second regions of the given real-world scene,respectively, thereby enabling the at least one warped image to becaptured. Notably, an optical portion having a larger focal lengthmagnifies a projection of the given region of the given real-world scenepassing therethrough to a greater extent as compared to another opticalportion having a smaller focal length. It will be appreciated that theprojections of the first and second regions of the given real-worldscene are significantly distorted upon being differently magnified viathe first and second optical portions, respectively. Upon beingdifferently magnified, the projections of the first and the secondregions of the given real-world scene produce at the camera the firstwarped portion and the second warped portion of the at least one warpedimage, respectively.

It will be appreciated that an angular resolution of the captured atleast one warped image is spatially-uniform. By “spatially-uniformangular resolution”, it is meant that the angular resolution of the atleast one warped image is uniform across an imaginary plane on which theat least one warped image is produced. Optionally, said image planecorresponds to the camera chip of the camera. Throughout the presentdisclosure, the term “angular resolution” of a given image refers to anumber of pixels per degree (namely, points per degree (PPD)) of anangular width of a given portion of the given image, wherein the angularwidth is measured from an imaginary point in a three-dimensional space.

By “warped”, it means that a given image would appear distorted ifviewed as such. Notably, a number of pixels employed to capture thesecond region of the given-real world scene using the imaging systemwill be more as compared to a number of pixels employed to capture thesecond region of the given-real world scene using an imaging systemwithout the optical element. However, the pixels corresponding to thecamera chip of the camera are uniformly spaced. Therefore, the at leastone warped image captured by the camera will have the spatially-uniformangular resolution.

Optionally, the first optical portion substantially surrounds the secondoptical portion, wherein a first focal length of the first opticalportion is smaller than a second focal length of the second opticalportion. As a result, the second optical portion magnifies theprojection of the second region of the given real-world scene passingtherethrough to a greater extent as compared to how the first opticalportion magnifies the projection of the first region of the givenreal-world scene passing therethrough. When the at least one warpedimage is de-warped to produce at least one de-warped image for beingdisplayed via the display apparatus, the first warped portioncorresponds to a first de-warped portion of the at least one de-warpedimage and the second warped portion corresponds to a second de-warpedportion of the at least one de-warped image. Notably, the firstde-warped portion forms a low-resolution area and the second de-warpedportion forms a high-resolution area of the at least one de-warpedimage. It will be appreciated that the at least one de-warped imageproduced for being displayed via the display apparatus has aspatially-variable angular resolution. By “spatially-variable angularresolution”, it is meant that an angular resolution of the at least onede-warped image varies spatially across an imaginary plane on which theat least one de-warped image is produced and/or incident.

While applying a de-warping effect, the first warped portion and thesecond warped portion of the at least one warped image would bedifferently magnified to produce the first de-warped portion and thesecond de-warped portion of the at least one de-warped image,respectively. Notably, the magnification effect provided whilstde-warping the at least one warped image is inverse of the magnificationeffect provided whilst capturing the at least one warped image.Specifically, the second warped portion is inversely magnified to agreater extent than the first warped portion. As a result, pixelscorresponding to the second de-warped portion would appear smaller andcloser than pixels corresponding to the first de-warped portion. Inother words, an angular resolution of the second de-warped portion isgreater than an angular resolution of the first de-warped portion.Therefore, the at least one de-warped image being displayed via thedisplay apparatus has the spatially-variable angular resolution.

Optionally, the optical element further comprises at least oneintermediary optical portion between the first optical portion and thesecond optical portion, the at least one intermediary optical portionhaving a focal length that is different from the first focal length andthe second focal length. As a result, the at least one intermediaryoptical portion has different depth of field and different opticalproperties with respect to magnification as compared to the firstoptical portion and the second optical portion. Optionally, a thirdfocal length of the at least one intermediary optical portion may behigher than the first focal length, but smaller than the second focallength.

Optionally, the at least one intermediary optical portion comprises asingle intermediary optical portion or a plurality of intermediaryoptical portions. Throughout the present disclosure, the term“intermediary optical portion” refers to a portion of the opticalelement that lies between the first optical portion and the secondoptical portion. In other words, an intermediary optical portion is aportion of the optical element that surrounds the second opticalportion, and is surrounded by the first optical portion.

Throughout the present disclosure, the term “means for adjusting theoptical focus” refers to a specialized equipment for adjusting theoptical focus of the at least one imaging unit. Notably, the means foradjusting the optical focus is employed to adjust at least one focusingparameter of the optical element to achieve a required optical focus forthe region of interest within the given real-world scene. When theregion of interest is “in focus”, a representation of said region withinthe at least one warped image appears extremely sharp. Alternatively,when the region of interest is “out of focus”, a representation of saidregion within the at least one warped image appears blurred. In the atleast one warped image, the region of interest is represented by thesecond warped portion, while a remaining region of the given real-worldscene is represented by the first warped portion.

Optionally, the means for adjusting the optical focus adjusts the focusof the at least one imaging unit by employing an active autofocusmechanism or a passive autofocus mechanism. The active autofocusmechanism is implemented using at least one of: an ultrasonic waveemitter, an infrared light emitter, a Light Detection and Ranging(LIDAR) camera, a Time-of-Flight (ToF) camera. Said active autofocusmechanism emits signals to measure distance to the at least one objectindependently, and subsequently adjust the optical element for achievinga required optical focus. The passive autofocus mechanism is implementedby a processing device that is configured to analyze phase and/orcontrast within at least one reference image of the given real-worldscene, and subsequently adjusts the optical element for achieving arequired optical focus. Such active autofocus mechanisms and passiveautofocus mechanisms are well-known in the art.

Throughout the present disclosure, the term “means for generating thedepth or the voxel map” refers to equipment and/or techniques configuredto record and represent optical depth (namely, optical distance) of thegiven real-world scene. Said means provides a frame of reference fromwhich the optical depth of any point within the given real-world scenecan be extracted.

Optionally, the means for generating the depth map or the voxel mapcomprises an imaging device configured to capture a depth image (namely,an image depicting depth) of the given real-world scene to generate thedepth map or the voxel map of the given real-world scene. Optionally, inthis regard, the depth image is a two-dimensional image or athree-dimensional image. Optionally, the captured depth image requiresfurther processing to accurately represent the optical depth of thegiven real-world scene. Furthermore, it will be appreciated that theimaging device could be a two-dimensional camera or a depth camera(namely, a ranging camera). Examples of the imaging device include, butare not limited to, a digital camera, an RGB-D camera, a LiDAR camera, aToF camera, a Sound Navigation and Ranging (SONAR) camera, a laserrangefinder, a stereo camera, a plenoptic camera, an infrared camera,and an ultrasound imaging equipment.

Additionally, optionally, the means for generating the depth map or thevoxel map comprises a processing module coupled to the imaging device,wherein the processing module is configured to process the captureddepth image for generating the depth map or the voxel map of the givenreal-world scene. In an example, the imaging device may be a stereocamera configured to capture a three-dimensional depth image of thegiven real-world scene. The processing module coupled to the stereocamera may process said depth image to create a disparity map that maybe employed to generate the depth map or the voxel map of the givenreal-world scene. In another example, the means for generating the depthmap or the voxel map may be implemented on a surveying device, whereinthe surveying device may be arranged to move within the real world scenefor (i) capturing the depth image of the given real-world scene usingthe imaging device, and (ii) employing Simultaneous Localization andMapping (SLAM) algorithm to process the captured depth image forgenerating the depth map or the voxel map of the given real-world scene.

Throughout the present disclosure, the term “depth map” relates to adata structure comprising information pertaining to the optical depth ofthe given real-world scene. Optionally, the depth map is an imagecomprising a plurality of pixels, wherein a color of each pixelindicates optical depth of its corresponding point(s) within the givenreal-world scene. As an example, the depth map may be a grayscale imagewherein each pixel is associated with a single monochromatic colorhaving intensity ranging from black color (namely, maximum intensity) towhite color (namely, minimum intensity), wherein a black colored-pixeldepicts maximum optical depth (namely, largest optical distance) of itscorresponding point within the given real-world scene, whilst a whitecolored pixel depicts minimum optical depth (namely, smallest opticaldistance) of its corresponding point within the given real-world scene.

Furthermore, throughout the present disclosure, the term “voxel map”used herein relates to a data structure comprising a plurality ofthree-dimensional volume elements that constitute the given real-worldscene, wherein each three-dimensional volume element represents athree-dimensional region within the given real-world scene. A giventhree-dimensional volume element is indicative of the optical depth ofits corresponding point(s) within the given real-world scene.

Optionally, the depth or voxel map is generated in real time.Alternatively, optionally, the depth or voxel map is generated a priori.In such a case, the means for generating generates (and optionally,analyses) the information pertaining to the optical depth of the givenreal-world scene at a specific time prior to capture of the at least onewarped image. Optionally, in this regard, the means for generatingupdates said information from time to time.

Throughout the present disclosure, the term “processor” refers tohardware, software, firmware or a combination of these. The processorcontrols operation of the imaging system. The processor is communicablycoupled to the at least one imaging unit and said means for generatingin a wireless manner and/or in a wired manner. By way of such coupling,the processor obtains the at least one image from the at least oneimaging unit. As an example, the imaging system may be mounted on arobot in a manner that the at least one imaging unit is mounted on anouter surface of the robot, whereas the means for generating and theprocessor are mounted inside a body of the robot. In such an example,the processor may be wirelessly coupled to the at least one imagingunit, and may be coupled via wires to the means for generating.

The processor is configured to obtain, from the display apparatus,information indicative of the current gaze direction of a user. Notably,the information indicative of the gaze direction of the user is obtainedby using the means for detecting the gaze direction, and thereafter,communicated from the display apparatus to the processor. The processoris at least coupled in communication with the display apparatus.

Throughout the present disclosure, the term “means for detecting thegaze direction” refers to specialized equipment for detecting and/orfollowing a direction of gaze of the user of the display apparatus.Notably, the gaze direction of the user is detected when the displayapparatus in operation is worn by the user. Optionally, the means fordetecting the gaze direction is implemented by way of contact lenseswith sensors, cameras monitoring the position of the pupil of the user'seye, and the like. Such means for detecting the gaze direction arewell-known in the art. Beneficially, the means for detecting the gazedirection is arranged in a manner that said means does not cause anyobstruction in an optical path of a projection of the at least onede-warped image (that is to be shown to the user). It is to beunderstood that the means for tracking the gaze direction may also bereferred to as an “eye-tracker system”, a “gaze-tracking system” or a“means for tracking the gaze direction”.

As an example, the means for detecting gaze direction may be implementedusing a set of illuminators for emitting light to illuminate the user'seye, a gaze-tracking camera for capturing an image of reflections of thelight from the user's eye, and a processing unit coupled incommunication with the set of illuminators and the gaze-tracking camera,wherein the processing unit is configured to detect the gaze directionof the user using the captured image.

Throughout the present disclosure, the term “de-warped image” refers toan image that is produced by applying the de-warping effect to the atleast one warped image. The imaging system produces the at least onewarped image, and components of the display apparatus apply thede-warping effect to the at least one warped image for producing the atleast one de-warped image. The de-warping effect is applied optically(for example, using at least one optical de-warping element), or viaimage processing. Said de-warping effect is an inverse of a warpingeffect that is provided by the optical element whilst capturing the atleast one warped image.

Optionally, a given image is displayed to the user via at least oneimage renderer of the display apparatus. Herein, the term “imagerenderer” refers to equipment that, in operation, renders the givenimage that is to be displayed to the user of the display apparatus. Thegiven image could be a warped image, or a de-warped image, depending onthe components and functionality of the display apparatus. Optionally,the at least one image renderer is implemented as at least one display.Optionally, the at least one image renderer is implemented as at leastone projector. In this regard, the given image is projected onto aprojection screen or directly onto a retina of the user's eyes.

Optionally, the image renderer is implemented as a Fovea ContingentDisplay (FCD), wherein the FCD comprises a first display having a firstdisplay resolution and a second display having a second displayresolution, the second display resolution being higher than the firstdisplay resolution. It will be appreciated that the FCD is designed toimitate a physiology of human vision. The FCD allows for increasingimmersion and realism within the simulated environment.

Optionally, the display apparatus further comprises an exit opticalelement. The term “exit optical element” refers to an optical componentthat is configured to direct a projection of the at least one de-warpedimage towards the user's eyes, when the display apparatus is worn by theuser. The term “exit optical element” is also commonly referred to as an“eyepiece”. Optionally, the exit optical element is implemented by wayof at least one of: a convex lens, a plano-convex lens, a Liquid Crystal(LC) lens, a liquid lens, a Fresnel lens, a spherical lens, a chromaticlens.

The processor is configured to determine, based on the gaze direction ofthe user and the depth or voxel map of the given real-world scene, theoptical depth of the at least one object present in the region ofinterest within the given real-world scene. Throughout the presentdisclosure, the term “region of interest” refers to a region of thegiven real-world scene whereat the gaze direction of the user's eyes arefocused at a given point of time. Notably, the region of interestcorresponds to the second region of the given real-world scene. It willbe appreciated that the region of interest is a fixation region withinthe given real-world scene. Therefore, the region of interest is aregion of focus of the user's gaze within the given real-world scene.Furthermore, it is to be understood that the region of interest relatesto a region resolved to a much greater detail as compared to otherregions of given real-world scene, when the given real-world scene isviewed by a human visual system (namely, by the user's eyes). Moreover,the at least one object present in a region of interest is a fixationobject within the given real-world scene. When the gaze direction of theuser is directed towards the at least one object, the at least oneobject is focused onto the fovea of the user's eyes, and is resolved toa much greater detail as compared to the remaining object(s) of thegiven real-world scene.

It will be appreciated that the “optical depth” of the at least oneobject present in the region of interest refers to an optical distancebetween said object and the camera.

Optionally, when determining the optical depth of the at least oneobject present in the region of interest, the processor is configured tomap a current gaze direction of the user to the depth or voxel map. Itwill be appreciated that “mapping the current gaze direction of the userto the depth or voxel map” refers to a process of associating thecurrent gaze direction of the user with the depth or the voxel map todetermine those data structure elements of the depth or the voxel mapthat substantially correspond to the region of interest within the givenreal-world scene. Thereafter, the processor extracts optical depthinformation associated with such data structure elements to determinethe optical depth of the at least one object.

The processor is configured to control the means for adjusting theoptical focus of the given imaging unit, based on the optical depth ofthe at least one object and the focal lengths of the first opticalportion and the second optical portion, to capture the at least onewarped image of the given real-world scene, the at least one warpedimage having the spatially-uniform angular resolution. The processorcontrols said means for adjusting the optical focus in a manner that theat least one object is “in focus” within the at least one warped image.Given the focal lengths of the first optical portion and the secondoptical portion, the optical element is required to be adjusted by themeans for adjusting the optical focus, in order to properly capture theat least one warped image. The processor controls the means foradjusting the optical focus to provide such required adjustment of theoptical element.

Optionally, when controlling the means for adjusting the optical focusof the given imaging unit, the processor is configured to adjust, basedon the gaze direction of the user, at least one focusing parameter ofthe optical element. Notably, different gaze directions correspond todifferent focal lengths of the optical element. A given focal lengthrequires a specific manner of adjusting focus. Therefore, for differentfocal lengths (of the optical element) corresponding to different gazedirections, the at least one focusing parameter is adjusted differently.

Optionally, the at least one focusing parameter of the optical elementis adjusted in a step-wise manner. Optionally, said adjustment is madeaccording to a Hill-climbing focusing algorithm. In such a case, aftereach step of adjustment, the processor checks whether the requiredoptical focus of the given imaging unit is achieved or not. If therequired optical focus of the given imaging unit is achieved, saidstep-wise adjustment is complete. If the required optical focus of thegiven imaging unit is not achieved, a subsequent step of the step-wiseadjustment is implemented.

Optionally, the at least one focusing parameter is at least one of: stepsize of a coarse focusing step, step size of a fine focusing step, stepsize of a return focusing step, a number of course focusing steps to beimplemented, a number of fine focusing steps to be implemented, a numberof return focusing steps to be implemented.

Moreover, optionally, the at least one focusing parameter is calculatedbased upon at least one of: a required blur value, a required final sizeof a circle of confusion, a focal length of the optical element, arequired full displacement of the optical element. It will beappreciated that different focal lengths require different focusingparameters. As an example, the step size of the coarse focusing step maybe calculated by using a 2 pixel size of the circle of confusion,whereas the step size of the fine focusing step may be calculated byusing a 1 pixel size or a 0.5 pixel size of the circle of confusion.

More optionally, the step size of at least one of: the coarse focusingstep, the fine focusing step, the return focusing step is calculatedbased upon:

-   -   the required full displacement of the optical element,    -   the number of at least one of: course focusing steps, fine        focusing steps, return focusing steps to be implemented, and    -   the required final size of the circle of confusion.

It will be appreciated that the aforesaid step size(s) is/are selectedin a manner that a resolution peak of the Hill-climbing focusingalgorithm is not missed while adjusting the optical focus of the givenimaging unit. The number of at least one of: course focusing steps, finefocusing steps, return focusing steps to be implemented is selected in amanner that said number is neither too many nor too less, therebyensuring that no resolution peak of the Hill-climbing focusing algorithmis missed.

For illustration purposes only, there will now be considered an examplewherein the required blur value B (associated with a given pixel pitchequal to 2 micrometers and a given size of the circle of confusion equalto 2 pixel size, or 4 micrometers) is equal to 4 micrometers, anaperture of the optical element is equal to 2.8 and the focal length ofthe optical element is equal to 2.5 millimeters. In such an example, thestep size of the coarse focusing step may be calculated using thefollowing mathematical formula:Step size=(2*B*Fno*f ²)/(f−B*Fno)²

wherein, ‘B’ represents the required blur value associated with thegiven pixel pitch and the given size of the circle of confusion, ‘Fno’represents the aperture of the optical element; and ‘F’ represents thefocal length of the optical element.

Upon substituting the example values in said formula, the step size ofthe coarse focusing step is calculated to be 22.60 micrometers.

Moreover, when a required full displacement (namely, the requiredadjustment) of the optical element is 31.65 micrometers, the number ofcoarse focusing steps to be implemented for the optical element is equalto 1.4 (notably, equal to 31.65/22.60 steps). Therefore, the means foradjusting the optical focus adjusts the optical element is byimplementing less than 2 coarse focusing steps to achieve the requiredfull displacement.

Optionally, the required full displacement of the optical element iscalculated based on the gaze direction of the user. Optionally, in thisregard, the required full displacement of the optical element iscalculated according to the focal length of the optical element and theoptical depth of the at least one object within the given real-worldscene. Optionally, said required full displacement of the opticalelement is calculated using the following mathematical formula:d=(((1/f)−(1/D))⁻¹ −f)×1000

wherein, ‘d’ represents the required full displacement of the opticalelement, ‘f’ represents the focal length of the optical element, and ‘D’represents the optical depth of the at least one object.

For illustration purposes only, there are now provided exemplarycalculations of the step size of the coarse focusing step and therequired full displacement of the optical element for various focallengths of the optical element in the table given below.

Angular distance 0 10 20 30 40 degrees degrees degrees degrees degreesFocal length (f) 6.5 5.5 4.5 3.5 2.5 in millimeters Number of coarse 9.76.9 4.6 2.8 1.4 focusing steps to achieve optical focus for D = 200millimeters Step size of 22.48 22.49 22.51 22.54 22.6 a coarse focusingstep in micrometers Required full 218.35 155.53 103.58 62.34 31.65displacement of the optical element (d) in micrometers for D = 200 mmStepwise 22.48 22.49 22.51 22.54 22.6 adjustment 44.96 44.98 45.02 45.0845.2 of the optical 67.44 67.47 67.53 67.62 element in 89.92 89.96 90.04micrometers 112.4 112.45 112.55 134.88 134.94 157.36 157.43 179.84202.32 224.8

Notably, in the aforementioned exemplary calculations, it will beappreciated that similar adjustment of the optical element for differentfocal lengths cause the optical element to focus at considerablydifferent optical depths. For example, upon 3 steps of adjustment of theoptical element having the focal lengths equal to 6.5 millimeters, 5.5millimeters, 4.5 millimeters, and 3.5 millimeters, the overalladjustment of the optical element would be 67.44 micrometers, 67.47micrometers, 67.53 micrometers, and 67.62 micrometers, respectively. Insuch a case,

-   -   when the optical element has the 6.5 millimeters focal length,        the optical element would focus at an optical depth of        approximately 630 millimeters,    -   when the optical element has the 5.5 millimeters focal length,        the optical element would focus at an optical depth of        approximately 450 millimeters,    -   when the optical element has the 4.5 millimeters focal length,        the optical element would focus at an optical depth of        approximately 304 millimeters, and    -   when the optical element has the 3.5 millimeters focal length,        the optical element would focus at an optical depth that is less        than 200 millimeters.

Optionally, the at least one object comprises a first object and asecond object, a first optical depth of the first object being differentfrom a second optical depth of the second object, wherein the processoris configured to:

-   -   select a given optical depth that lies between the first optical        depth and the second optical depth; and    -   adjust the optical focus of the given imaging unit, based on the        given optical depth, to capture the at least one warped image of        the given real-world scene.

Optionally, the given optical depth is selected in a manner that thefirst optical depth and the second optical depth lie within a depth offield corresponding to the given optical depth. Given the focal lengthsof the first and second optical portions, the processor adjusts theoptical focus of the given imaging unit in a manner that a region of thegiven real-world scene that lies at the given optical depth is focusedsharply onto the camera. Moreover, since the first optical depth and thesecond optical depth lie within the depth of field, an entire region ofthe given real-world scene that lies between the first optical depth andthe second optical depth would be focused onto the camera with anacceptable sharpness. Thus, both the first object and the second objectwould appear sharp in the at least one warped image, despite being atdifferent optical depths. It will be appreciated that by focusing at thegiven optical depth and utilizing the depth of field, a single imagingunit can capture the at least one warped image in a manner that a rangeof optical depths within the given real-world scene is captured withacceptable sharpness. Remaining regions of the given real-world scenethat lie outside of said range of optical depths would appear blurredwithin the at least one warped image.

Optionally, the imaging system comprises separate imaging unitscorresponding to a left eye and a right eye of the user. In such a case,separate depth or voxel maps are generated from a perspective of theleft eye and a perspective of the right eye of the user. Therefore, whenthe first optical depth of the first object is different from the secondoptical depth of the second object, separate given optical depths areselected for the imaging units corresponding to the left eye and theright eye. Therefore, the optical focus is adjusted differently for boththe separate imaging units, based on the separate given optical depths,to capture at least one left-perspective warped image and at least oneright-perspective warped image of the given real-world scene. It will beappreciated that the at least one left-perspective warped image and atleast one right-perspective warped image are offset from each other. Asa result, when the at least one left-perspective warped image and atleast one right-perspective warped image are de-warped and shown to theuser via the display apparatus, the user would experience considerablerealism and immersion within the visual scene, by accurately perceivingstereoscopic depth and focus within the visual scene.

Alternatively, optionally, the at least one object comprises a firstobject and a second object, a first optical depth of the first objectbeing different from a second optical depth of the second object, the atleast one imaging unit comprising a first imaging unit and a secondimaging unit, wherein the processor is configured to adjust an opticalfocus of the first imaging unit and an optical focus of the secondimaging unit, based on the first optical depth and the second opticaldepth, to capture a first warped image and a second warped image of thegiven real-world scene, respectively. In such a case, the first imagingunit is made to focus at the first optical depth and the second imagingunit is made to focus at the second optical depth. As a result, in thefirst warped image, the first object appears extremely sharp. Likewise,in the second warped image, the second object appears extremely sharp.Both the first and second warped images can therefore be utilized at thedisplay apparatus to display sharp representations of the first andsecond objects to the user.

It will be appreciated that using the first and second imaging units tofocus at the first and second optical depths is especially useful whenthe first and second objects lie at the first and second optical depths,respectively, along the gaze direction of the user. In such a case, theuser could be looking at either the first object or second object, sinceboth objects lie along his/her gaze direction. By using the first andsecond imaging units, the imaging system focuses sharply at both thefirst and second optical depths, in order to capture both said objectssharply. Therefore, when the first and second warped images are utilizedat the display apparatus to present the visual scene to the user, boththe first and second objects have acceptable sharpness.

Optionally, at the display apparatus, the first warped image is shown tothe left eye of the user and the second warped image is shown to theright eye of the user. Since both the first and second warped images arecaptured using different imaging units, they are offset with respect toeach other. When the first and second warped images are shown to theuser, the user correctly perceives different optical depths due tohis/her stereoscopic vision, whilst also perceiving sharpness of thefirst and second objects at the first and second optical depths.

Optionally, the means for adjusting the optical focus of the givenimaging unit comprises at least one first actuator that, in operation,moves the optical element along an optical axis of the camera of thegiven imaging unit. In such a case, the at least one first actuatorprovides a translational motion of the optical element along the opticalaxis of the camera. By way of such movement, the optical element ismoved closer to or away from the camera. This, in turn, changes how theat least one object is focused at the camera (more specifically, at thephotosensitive surface of the camera chip). It will be appreciated thatthe at least one first actuator moves the optical element to a positionalong the optical axis of the camera at which a sharpest possible focusof the at least one object is achieved. Moreover, the at least one firstactuator could be directly coupled or indirectly coupled (for example,via another component) to the optical element.

Throughout the present disclosure, the term “actuator” refers toequipment (for example, such as electrical components, mechanicalcomponents, magnetic components, polymeric components, and so forth)that is employed to move its associated component. Optionally, a givenactuator moves its associated component using an actuation signal (forexample, such as an electric current, hydraulic pressure, and the like).More optionally, the processor controls the given actuator by generatingthe actuation signal.

Optionally, the means for adjusting the optical focus of the givenimaging unit comprises a focusing optical element and at least onesecond actuator that, in operation, moves the focusing optical elementalong an optical axis of the camera. Optionally, in this regard, thefocusing optical element is positioned on the optical path between theoptical element and the camera of the given imaging unit. Herein, theterm “focusing element” refers to an optical component that isspecifically employed for purposes of adjusting the optical focus of thegiven imaging unit. Optical properties of the focusing optical element,in combination with optical properties of the optical element, provide arequired optical focus of the given imaging unit. The focusing opticalelement is moved closer to or away from the optical element, along theoptical axis of the camera. This changes a separation between thefocusing optical element and the optical element. As a result, theoptical focus of the given imaging unit also changes. It will beappreciated that the at least one second actuator moves the focusingoptical element to a position along the optical axis of the camerawhereat a separation between the focusing optical element and theoptical element is suitable to achieve a sharpest possible focus of theat least one object. Moreover, the at least one second actuator could bedirectly coupled or indirectly coupled (for example, via anothercomponent) to the focusing optical element.

Optionally, the focusing element is implemented as at least one of: aplano-convex lens, a biconvex lens, a plano-concave lens, a biconcavelens, an aspheric lens, a Fresnel lens. Optionally, when the focusingelement is implemented as a plurality of the aforesaid opticalsub-components, at least one of the plurality of optical sub-componentsis movable using the at least one second actuator. As an example, thefocusing element may be implemented as a configuration of 5 biconvexlenses, wherein the at least one second actuator may move 3 biconvexlenses among the 5 biconvex lenses along an optical axis of the camera.

Optionally, the means for adjusting the optical focus of the givenimaging unit comprises an active focusing optical element, wherein theprocessor is configured to adjust an active optical characteristic ofthe active focusing optical element. Examples of said active opticalcharacteristic include, but are not limited to, a focal length and arefractive index. Optionally, in this regard, the active focusingoptical element is implemented as one of: a liquid crystal lens, aliquid lens, a polymer lens. The focal length of the active focusingoptical element is adjusted by: changing curvature of the activefocusing optical element, changing orientation of molecules of an activemedium of the active focusing optical element by varying the electricfield across said active focusing optical element, and the like. As anexample, the active focusing optical element may be implemented as thepolymer lens, wherein the processor adjusts the focal length of thepolymer lens by controlling a micromechanical actuator coupled to thepolymer lens to physically press against the polymer lens for changingthe curvature of the polymer lens.

Optionally, the optical element and the means for adjusting the opticalfocus of the given imaging unit are implemented together as adynamically-controllable optical element, the focal lengths of the firstoptical portion and the second optical portion of the optical elementbeing dynamically changeable. In this regard, optical properties of thedynamically-controllable optical element can be adjusted without movingit. Optionally, the processor is configured to dynamically change thefocal lengths of the first optical portion and the second opticalportion of the dynamically-controllable optical element via a dynamiccontrol signal. Optionally, in this regard, the dynamic control signalis at least one of: an electrical signal, a mechanical signal, a lightsignal, a thermal signal.

In an embodiment, the dynamic control signal is employed to adjust acurvature of the dynamically-controllable optical element. Optionally,in this regard, the dynamically-controllable optical element is made ofan electrically controllable an active polymer or a flexible membranematerial. Upon a change in curvature of the dynamically-controllableoptical element, the focal lengths of the first optical portion and thesecond optical portion would also change. Such a change in focal lengthsleads to a corresponding change in focus of the given imaging unit.Therefore, by adjusting such a dynamic control signal, the curvature ofthe dynamically-controllable optical element is adjusted in a mannerthat required focal lengths of the first optical portion and the secondoptical portion are provided for achieving a sharpest possible focus ofthe at least one object.

Optionally, the dynamically-controllable optical element is implementedas one of: a fluid lens, a liquid crystal lens, a polymer lens, a mirrorwhose curvature can be changed dynamically.

In another embodiment, the dynamic control signal is employed to adjustan active optical characteristic curvature of thedynamically-controllable optical element. In this regard, thedynamically-controllable optical element contains an active medium (forexample, such as liquid crystals) that is controllable to adjust thefocal lengths of the first optical portion and the second opticalportion. Said active medium is controlled in a manner that requiredfocal lengths of the first optical portion and the second opticalportion are provided for achieving a sharpest possible focus of the atleast one object.

Optionally, the dynamically-controllable optical element is implementedas one of: a liquid crystal lens, a liquid lens.

Optionally, the optical element is rotationally asymmetric, the givenimaging unit comprising at least one third actuator associated with theoptical element, wherein the processor is configured to control the atleast one third actuator to adjust a rotational orientation of theoptical element according to the gaze direction of the user. Optionally,in such a case, the optical element is rotated (notably, about itscenter of rotation with respect to the camera. Optionally, the opticalelement is rotated to cover a circular area on the camera chip of thecamera. In particular, the rotational orientation of the optical elementis adjusted by the third actuator in a manner that the projection of thesecond region of the given real-world scene passes through or reflectsfrom the second optical portion, whereas the projection of the firstregion of the given real-world scene passes through or reflects from thefirst optical portion. With a change in the gaze direction of the user,the first and second regions of the given real-world scene would alsochange, and therefore, the rotational orientation of the optical elementwould also be changed. The optical element is rotated to a givenposition, and the rotation is stopped when the optical element isaligned according to the detected gaze direction. In this way, theoptical element is rotated repeatedly, based upon the detected gazedirection.

In some implementations, the optical element is asymmetrical about itsoptical axis. In such implementations, the optical element would alwaysbe rotationally asymmetric. In other implementations, the opticalelement is symmetrical about its optical axis. In such implementations,the optical element may or may not be rotationally asymmetric.

Optionally, when the optical element is asymmetrical about its opticalaxis,

-   -   if the optical element is rotatable in only one direction, an        angle of rotation of the optical element lies within a range of        0 degrees to 360 degrees; otherwise,    -   if the optical element is rotatable in both the directions, the        angle of rotation of the optical element lies within a range of        0 degrees to 180 degrees. One such example implementation has        been illustrated in conjunction with FIG. 11.

Optionally, when the optical element is symmetrical about its opticalaxis and is rotationally asymmetric,

-   -   if the optical element is rotatable in only one direction, the        angle of rotation of the optical element lies within a range of        0 degrees to 180 degrees; otherwise,    -   if the optical element is rotatable in both the directions, the        angle of rotation of the optical element lies within a range of        0 degrees to 90 degrees. One such example implementation has        been illustrated in conjunction with FIG. 10.

It will be appreciated that angle of rotation of the optical element isreduced considerably in a case where the optical element is symmetricalas compared to another case where the optical element is asymmetrical.As a result, the at least one third actuator is simpler to implement fora symmetrical optical element as compared to an asymmetrical opticalelement. Moreover, power consumption of the at least one third actuatoralso reduces in the case where the at least one optical element issymmetrical.

It will be appreciated that the optical center of the optical elementmay or may not be the same as a center of rotation. Moreover, it will beappreciated that the optical element is balanced in a manner that acenter of mass of the optical element is at the center of rotation.

The present disclosure also relates to the method as described above.Various embodiments and variants disclosed above apply mutatis mutandisto the method.

Optionally, in the method, the at least one object comprises a firstobject and a second object, a first optical depth of the first objectbeing different from a second optical depth of the second object,wherein the method further comprises:

-   -   selecting a given optical depth that lies between the first        optical depth and the second optical depth; and    -   adjusting the optical focus of the given imaging unit, based on        the given optical depth, to capture the at least one warped        image of the given real-world scene.

Alternatively, optionally, in the method, the at least one objectcomprises a first object and a second object, a first optical depth ofthe first object being different from a second optical depth of thesecond object, the at least one imaging unit comprising a first imagingunit and a second imaging unit, wherein the method further comprisesadjusting an optical focus of the first imaging unit and an opticalfocus of the second imaging unit, based on the first optical depth andthe second optical depth, to capture a first warped image and a secondwarped image of the given real-world scene, respectively.

Optionally, in the method, the means for adjusting the optical focus ofthe given imaging unit comprises at least one first actuator associatedwith the optical element, wherein the step of adjusting the opticalfocus comprises moving, via the at least one first actuator, the opticalelement along an optical axis of the camera.

Optionally, in the method, the means for adjusting the optical focus ofthe given imaging unit comprises a focusing optical element and at leastone second actuator associated therewith, wherein the step of adjustingthe optical focus comprises moving, via the at least one secondactuator, the focusing optical element along an optical axis of thecamera.

Optionally, in the method, the optical element and the means foradjusting the optical focus of the given imaging unit are implementedtogether as a dynamically-controllable optical element, wherein themethod further comprises dynamically changing the focal lengths of thefirst optical portion and the second optical portion of the opticalelement.

Optionally, in the method, the step of adjusting the optical focus ofthe given imaging unit comprises adjusting, based on the gaze directionof the user, at least one focusing parameter of the optical element.

Optionally, in the method, the first optical portion substantiallysurrounds the second optical portion, wherein a first focal length ofthe first optical portion is smaller than a second focal length of thesecond optical portion.

Optionally, in the method, the optical element is rotationallyasymmetric, the given imaging unit comprising at least one thirdactuator associated with the optical element, wherein the method furthercomprises controlling the at least one third actuator to adjust arotational orientation of the optical element according to the gazedirection of the user.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring to FIG. 1, illustrated is a block diagram of architecture ofan imaging system 100 for producing images for a display apparatus 102,in accordance with an embodiment of the present disclosure. The imagingsystem 100 comprises at least one imaging unit (depicted as an imagingunit 104), means 106 for generating a depth or voxel map of a givenreal-world scene, and a processor 108 communicably coupled to theimaging unit 104 and said means 106 for generating. The imaging unit 104comprises a camera 110, an optical element 112 that comprises at least afirst optical portion and a second optical portion having differentfocal lengths, and means 114 for adjusting an optical focus of theimaging unit 104. The processor 108 is configured to obtain, from thedisplay apparatus 102, information indicative of a gaze direction of auser; determine, based on the gaze direction of the user and the depthor voxel map of the given real-world scene, an optical depth of at leastone object present in a region of interest within the given real-worldscene; and control the means 114 for adjusting the optical focus of theimaging unit 104, based on the optical depth of the at least one objectand the focal lengths of the first optical portion and the secondoptical portion, to capture at least one warped image of the givenreal-world scene, the at least one warped image having aspatially-uniform angular resolution.

Referring to FIG. 2, illustrated is a block diagram of architecture ofan imaging system 200 for producing images for a display apparatus 202,in accordance with another embodiment of the present disclosure. Theimaging system 200 comprises at least one imaging unit (depicted as animaging unit 204), means 206 for generating a depth or voxel map of agiven real-world scene, and a processor 208 communicably coupled to theimaging unit 204 and said means 206 for generating. The imaging unit 204comprises a camera 210, an optical element 212, and means 214 foradjusting an optical focus of the imaging unit 204.

As shown, the means 214 for adjusting the optical focus of the imagingunit 204 comprises at least one first actuator (depicted as a firstactuator 216) that, in operation, moves the optical element 212 along anoptical axis of the camera 210.

Moreover, the means 214 for adjusting the optical focus of the imagingunit 204 further comprises a focusing optical element 218 and at leastone second actuator (depicted as a second actuator 220) that, inoperation, moves the focusing optical element 218 along an optical axisof the camera 210.

Referring to FIG. 3, illustrated is a block diagram of architecture ofan imaging system 300 for producing images for a display apparatus 302,in accordance with yet another embodiment of the present disclosure. Theimaging system 300 comprises at least one imaging unit (depicted as animaging unit 304), means 306 for generating a depth or voxel map of agiven real-world scene, and a processor 308 communicably coupled to theimaging unit 304 and said means 306 for generating.

The imaging unit 304 comprises a camera 310, an optical element 312 thatcomprises at least a first optical portion and a second optical portionhaving different focal lengths, and means 314 for adjusting an opticalfocus of the imaging unit 304. When the optical element 312 isrotationally asymmetric, the imaging unit 304 further comprises at leastone third actuator (depicted as a third actuator 316) associated withthe optical element 312, wherein the processor 308 is configured tocontrol the third actuator 316 to adjust a rotational orientation of theoptical element 312 according to the gaze direction of the user.

It may be understood by a person skilled in the art that FIG. 1, FIG. 2,and FIG. 3 depict simplified block diagrams of architectures of theimaging systems 100, 200, and 300, respectively, for sake of clarity,which should not unduly limit the scope of the claims herein. The personskilled in the art will recognize many variations, alternatives, andmodifications of embodiments of the present disclosure.

Referring to FIG. 4, illustrated is an exemplary real-world scene 400including two objects 402 and 404 at different optical depths, said twoobjects 402 and 404 being captured using a single imaging unit 406, inaccordance with an embodiment of the present disclosure. In this regard,the object 402 can be considered as a first object, whereas the object404 can be considered as a second object. As shown, a first opticaldepth D1 of the first object 402 is different from a second opticaldepth D3 of the second object 404. In the real-world scene 400, thefirst optical depth D1 is lesser than the second optical depth D3. Insuch a case, a processor of an imaging system is configured to select agiven optical depth D2 that lies between the first optical depth D1 andthe second optical depth D3, and adjust an optical focus of the imagingunit 406 of the imaging system, based on the given optical depth D2, tocapture at least one warped image of the real-world scene 400.

It may be understood by a person skilled in the art that FIG. 4 depictsa simplified illustration of the exemplary real-world scene 400 and theimaging unit 406 for sake of clarity, which should not unduly limit thescope of the claims herein. The person skilled in the art will recognizemany variations, alternatives, and modifications of embodiments of thepresent disclosure.

Referring to FIG. 5, illustrated is an exemplary real-world scene 500including two objects 502 and 504 at different optical depths, said twoobjects 502 and 504 being captured using two imaging units 506 and 508,in accordance with an embodiment of the present disclosure. In thisregard, the object 502 can be considered as a first object, whereas theobject 504 can be considered as a second object. As shown, a firstoptical depth X1 of the first object 502 is different from a secondoptical depth X2 of the second object 504. In the real-world scene 500,the first optical depth X1 is lesser than the second optical depth X2.Moreover, the imaging unit 506 can be considered as a first imagingunit, whereas the imaging 508 can be considered as a second imagingunit. In such a case, a processor of an imaging system is configured toadjust an optical focus of the first imaging unit 506 and an opticalfocus of the second imaging unit 508, based on the first optical depthX1 and the second optical depth X2, to capture a first warped image anda second warped image of the real-world scene 500, respectively.Notably, the first imaging unit 506 is focused at the first opticaldepth X1, whereas the second imaging unit 508 is focused at the secondoptical depth X2.

It may be understood by a person skilled in the art that FIG. 5 depictsa simplified illustration of the exemplary real-world scene 500 and theimaging units 506 and 508 for sake of clarity, which should not undulylimit the scope of the claims herein. The person skilled in the art willrecognize many variations, alternatives, and modifications ofembodiments of the present disclosure.

Referring to FIG. 6, illustrated is an example graphical representationof depth of field of a given imaging unit as a function of focal lengthof an optical element of the given imaging unit, in accordance with anembodiment of the present disclosure. With reference to FIG. 6, theoptical element has variable optical properties across its field ofview. As shown, the focal length of the optical element is maximum alongan axis passing through the center of the second optical portion, anddecreases on going away from said center. As an example, the focallength of the optical element along the axis passing through the centerof the second optical portion is f1, whereas the focal length of theoptical element at an angular distance of 30 degrees (depicted as θ)from said axis is f2, wherein f2 is lesser than f1.

As a result, the depth of field of the given imaging unit variesinversely with respect to the focal length of the optical element of thegiven imaging unit. When the focal length of the optical element is f1(for example, equal to 7 millimeters), a narrow depth of field A1 isprovided by the given imaging unit. When the focal length of the opticalelement is f2 (for example, equal to 5 millimeters), a wide depth offield A2 is provided by the given imaging unit.

It may be understood by a person skilled in the art that FIG. 6 depictsan exemplary graphical representation for sake of clarity, which shouldnot unduly limit the scope of the claims herein. The person skilled inthe art will recognize many variations, alternatives, and modificationsof embodiments of the present disclosure.

Referring to FIG. 7, illustrated is an example graphical representationof how focus is adjusted via an imaging system, in accordance with anembodiment of the present disclosure. As shown, the optical focus of agiven imaging unit of the imaging system is adjusted in a step-wisemanner, according to a Hill-climbing focusing algorithm. According tothe step-wise manner, the optical focus of the given imaging unit isadjusted as a combination of coarse steps (depicted as solid steps) andfine steps (depicted as dashed steps) until a required optical focus isobtained. Notably, step size of the coarse steps and the fine steps iscalculated by the processor of the imaging system, based on opticalparameters of the imaging system and the gaze direction of the user. Asshown, the step size of the coarse steps is greater than the step sizeof the fine steps. Moreover, when a contrast of a currently capturedimage is lower than a contrast of a previously captured image duringrepetitive coarse stepping, a resolution peak of the Hill-climbingfocusing algorithm is understood to be missed. In such a case,repetitive fine stepping is employed to reach the resolution peak of theHill-climbing focusing algorithm. A return step (depicted as a dottedstep) indicates a step difference between an end point of the repetitivecoarse stepping and an end point of the repetitive fine stepping. In theexample graphical representation, the return step has a larger step sizeas compared to the coarse steps.

It may be understood by a person skilled in the art that FIG. 7 depictsan exemplary graphical representation for sake of clarity, which shouldnot unduly limit the scope of the claims herein. The person skilled inthe art will recognize many variations, alternatives, and modificationsof embodiments of the present disclosure.

Referring to FIG. 8, illustrated is an example implementation of adisplay apparatus 800, in accordance with an embodiment of the presentdisclosure. The display apparatus 800, in operation, is worn by a useron his/her head. The display apparatus 800 comprises at least one imagerenderer (depicted as an image renderer 802), a means 804 for detectinggaze direction, an exit optical element 806, and a processing module(not shown). The image renderer 802, in operation, renders an image. Inthis example implementation, the image is optionally a de-warped image.A projection of the de-warped image passes through the exit opticalelement 806, to be incident upon the user's eye. The means 804 fordetecting gaze direction is implemented using a set of illuminators(depicted as illuminators 808 and 810) for emitting light to illuminatethe user's eye, a gaze-tracking camera 812 for capturing an image ofreflections of the light from the user's eye, and a processing unit (notshown) coupled in communication with the set of illuminators 808 and 810and the gaze-tracking camera 812, wherein the processing unit isconfigured to detect the gaze direction of the user using the capturedimage.

Referring to FIG. 9, illustrated is an example implementation of animaging unit 900, in accordance with an embodiment of the presentdisclosure. The imaging unit 900 comprises a camera 902, an opticalelement, and a means for adjusting an optical focus of the imaging unit.An optical axis of the imaging unit 900 is represented as a long-dashedline X-X′. The camera 902 comprises at least an image sensor 904. Theoptical element comprises at least a first optical portion and a secondoptical portion having different focal lengths. A projection of a givenreal-world scene is differently magnified by the first optical portionand the second optical portion. As shown, a first region of theprojection of a given real-world scene (depicted as solid lines) ismagnified by the first optical portion, whereas a second region of theprojection of the given real-world scene (depicted as small-dashedlines) is de-magnified by the second optical portion.

Moreover, in the imaging unit 900, the optical element and the means foradjusting the optical focus of the given imaging unit are implementedtogether as a dynamically-controllable optical element 906, the focallengths of the first optical portion and the second optical portion ofthe optical element being dynamically changeable. A processor of animaging system is configured to control the dynamically-controllableoptical element 906 to adjust an optical focus of the imaging unit 900at a given focal plane FP within the given real-world scene.

FIG. 9 is merely an example, which should not unduly limit the scope ofthe claims herein. A person skilled in the art will recognize manyvariations, alternatives, and modifications of embodiments of thepresent disclosure. For example, some regions of the projection of thegiven real-world scene may be neither magnified nor de-magnified.

Referring to FIG. 10, illustrated is a schematic illustration of anexample implementation where a symmetrical optical element 1002 isrotated with respect to a camera, in accordance with an embodiment ofthe present disclosure. In this example implementation, the opticalelement 1002 is symmetrical about its optical axis and a second opticalportion 1004 is substantially ellipsoidal in shape. A first opticalportion 1006 substantially surrounds the second optical portion 1004,wherein a first focal length of the first optical portion 1006 issmaller than a second focal length of the second optical portion 1004.

In FIG. 10, there is shown a centre (depicted by a black dot) of thesecond optical portion 1004, which is also a centre of rotation. Twolines representing X and Y directions pass through the centre ofrotation, which overlaps with the centre of a warped image. The opticalelement 1002 is rotated (namely, about the centre of rotation) to covera circular area 1008 on a camera chip 1010 of the camera using thesecond optical portion 1004.

The optical element 1002 is rotated to a given position, and therotation is stopped when the second optical portion 1004 is alignedaccording to the detected gaze direction. In this way, the opticalelement 1002 is rotated repeatedly, based upon the detected gazedirection. A symmetrical optical element such as the optical element1002 may or may not be rotationally asymmetric.

When moving from a first position to a second position (namely, withrespect to a change in the user's gaze direction), the optical element1002 is required to be rotated at an angle that lies in:

-   -   a range of 0 degrees to 180 degrees, when the optical element        1002 rotates in only one direction, or    -   a range of 0 degrees to 90 degrees, when the optical element        1002 rotates in both directions.

Referring to FIG. 11, illustrated is a schematic illustration of anotherexample implementation where an asymmetrical optical element 1102 isrotated with respect to a camera, in accordance with another embodimentof the present disclosure. In this example implementation, the opticalelement 1102 is asymmetrical about its optical axis and a second opticalportion 1104 is substantially ellipsoidal in shape. A first opticalportion 1106 substantially surrounds the second optical portion 1104,wherein a first focal length of the first optical portion 1106 issmaller than a second focal length of the second optical portion 1104.

In FIG. 11, there are shown a centre ‘O’ of the second optical portion1104 and a centre of rotation (depicted by a black dot). Two linesrepresenting X′ and Y′ directions pass through the centre of rotation,which overlaps with the centre of a warped image. As the optical centre‘O’ of the second optical portion 1104 is not the same as the centre ofrotation, the optical element 1102 is rotated (namely, about the centreof rotation) to cover a circular area 1108 on a camera chip 1110 of thecamera using the second optical portion 1104.

The optical element 1102 is rotated to a given position, and therotation is stopped when the second optical portion 1104 is alignedaccording to the detected gaze direction. In this way, the opticalelement 1102 is rotated repeatedly, based upon the detected gazedirection. An asymmetrical optical element such as the optical element1102 is rotationally asymmetric.

When moving from a first position to a second position (namely, withrespect to a change in the user's gaze direction), the optical element1102 is required to be rotated at an angle that lies in:

-   -   a range of 0 degrees to 360 degrees, when the optical element        1102 rotates in only one direction, or    -   a range of 0 degrees to 180 degrees, when the optical element        1102 rotates in both directions.

FIGS. 10 and 11 are merely examples, which should not unduly limit thescope of the claims herein. A person skilled in the art will recognizemany variations, alternatives, and modifications of embodiments of thepresent disclosure. It will be appreciated that the optical elements1002 and 1102 have been depicted as lenses, for the sake of convenienceonly; the optical elements 1002 and 1102 are not limited to a particulartype of optical element. In other words, the optical elements 1002 and1102 can be implemented as a single lens or mirror having a complexshape or as a configuration of lenses and/or mirrors.

Referring to FIG. 12, illustrated are steps of a method for producingimages for a display apparatus, in accordance with an embodiment of thepresent disclosure. The method is implemented via an imaging systemcomprising at least one imaging unit, a given imaging unit comprising acamera, an optical element that comprises at least a first opticalportion and a second optical portion having different focal lengths, andmeans for adjusting an optical focus of the given imaging unit. At astep 1202, information indicative of a gaze direction of a user isobtained from the display apparatus. At a step 1204, a depth or voxelmap of a given real-world scene is generated. At a step 1206, an opticaldepth of at least one object present in a region of interest within thegiven real-world scene is determined based on the gaze direction of theuser and the depth or voxel map of the given real-world scene. At astep, 1208, an optical focus of the given imaging unit is adjusted basedon the optical depth of the at least one object and the focal lengths ofthe first optical portion and the second optical portion, to capture atleast one warped image of the given real-world scene. The at least onewarped image having a spatially-uniform angular resolution.

The steps 1202 to 1208 are only illustrative and other alternatives canalso be provided where one or more steps are added, one or more stepsare removed, or one or more steps are provided in a different sequencewithout departing from the scope of the claims herein.

Modifications to embodiments of the present disclosure described in theforegoing are possible without departing from the scope of the presentdisclosure as defined by the accompanying claims. Expressions such as“including”, “comprising”, “incorporating”, “have”, “is” used todescribe and claim the present disclosure are intended to be construedin a non-exclusive manner, namely allowing for items, components orelements not explicitly described also to be present. Reference to thesingular is also to be construed to relate to the plural.

What is claimed is:
 1. An imaging system for producing images for adisplay apparatus, the imaging system comprising: at least one imagingunit, a given imaging unit comprising: a camera; an optical element thatcomprises at least a first optical portion and a second optical portionhaving different focal lengths; and means for adjusting an optical focusof the given imaging unit; means for generating a depth or voxel map ofa given real-world scene; and a processor communicably coupled to the atleast one imaging unit and said means for generating, wherein theprocessor is configured to: obtain, from the display apparatus,information indicative of a gaze direction of a user; determine, basedon the gaze direction of the user and the depth or voxel map of thegiven real-world scene, an optical depth of at least one object presentin a region of interest within the given real-world scene; and controlthe means for adjusting the optical focus of the given imaging unit,based on the optical depth of the at least one object and the focallengths of the first optical portion and the second optical portion, tocapture at least one warped image of the given real-world scene, the atleast one warped image having a spatially-uniform angular resolutions;wherein the at least one object comprises a first object and a secondobject, a first optical depth of the first object being different from asecond optical depth of the second object, wherein the processor isconfigured to: select a given optical depth that lies between the firstoptical depth and the second optical depth; and adjust the optical focusof the given imaging unit, based on the given optical depth, to capturethe at least one warped image of the given real-world scene.
 2. Theimaging system of claim 1, wherein the at least one object comprises afirst object and a second object, a first optical depth of the firstobject being different from a second optical depth of the second object,the at least one imaging unit comprising a first imaging unit and asecond imaging unit, wherein the processor is configured to adjust anoptical focus of the first imaging unit and an optical focus of thesecond imaging unit, based on the first optical depth and the secondoptical depth, to capture a first warped image and a second warped imageof the given real-world scene, respectively.
 3. The imaging system ofclaim 1, wherein the means for adjusting the optical focus of the givenimaging unit comprises at least one first actuator that, in operation,moves the optical element along an optical axis of the camera.
 4. Theimaging system of claim 1, wherein the means for adjusting the opticalfocus of the given imaging unit comprises a focusing optical element andat least one second actuator that, in operation, moves the focusingoptical element along an optical axis of the camera.
 5. The imagingsystem of claim 1, wherein the optical element and the means foradjusting the optical focus of the given imaging unit are implementedtogether as a dynamically-controllable optical element, the focallengths of the first optical portion and the second optical portion ofthe optical element being dynamically changeable.
 6. The imaging systemof claim 1, wherein when controlling the means for adjusting the opticalfocus of the given imaging unit, the processor is configured to adjust,based on the gaze direction of the user, at least one focusing parameterof the optical element.
 7. The imaging system of claim 1, wherein thefirst optical portion substantially surrounds the second opticalportion, wherein a first focal length of the first optical portion issmaller than a second focal length of the second optical portion.
 8. Theimaging system of claim 1, wherein the optical element is rotationallyasymmetric, the given imaging unit comprising at least one thirdactuator associated with the optical element, wherein the processor isconfigured to control the at least one third actuator to adjust arotational orientation of the optical element according to the gazedirection of the user.
 9. A method for producing images for a displayapparatus, the method being implemented via an imaging system comprisingat least one imaging unit, a given imaging unit comprising a camera, anoptical element that comprises at least a first optical portion and asecond optical portion having different focal lengths, and means foradjusting an optical focus of the given imaging unit, the methodcomprising: obtaining, from the display apparatus, informationindicative of a gaze direction of a user; generating a depth or voxelmap of a given real-world scene; determining, based on the gazedirection of the user and the depth or voxel map of the given real-worldscene, an optical depth of at least one object present in a region ofinterest within the given real-world scene; and adjusting an opticalfocus of the given imaging unit, based on the optical depth of the atleast one object and the focal lengths of the first optical portion andthe second optical portion, to capture at least one warped image of thegiven real-world scene, the at least one warped image having aspatially-uniform angular resolution, wherein the at least one objectcomprises a first object and a second object, a first optical depth ofthe first object being different from a second optical depth of thesecond object, wherein the method further comprises: selecting a givenoptical depth that lies between the first optical depth and the secondoptical depth; and adjusting the optical focus of the given imagingunit, based on the given optical depth, to capture the at least onewarped image of the given real-world scene.
 10. The method of claim 9,wherein the at least one object comprises a first object and a secondobject, a first optical depth of the first object being different from asecond optical depth of the second object, the at least one imaging unitcomprising a first imaging unit and a second imaging unit, wherein themethod further comprises adjusting an optical focus of the first imagingunit and an optical focus of the second imaging unit, based on the firstoptical depth and the second optical depth, to capture a first warpedimage and a second warped image of the given real-world scene,respectively.
 11. The method of claim 9, wherein the means for adjustingthe optical focus of the given imaging unit comprises at least one firstactuator associated with the optical element, wherein the step ofadjusting the optical focus comprises moving, via the at least one firstactuator, the optical element along an optical axis of the camera. 12.The method of claim 9, wherein the means for adjusting the optical focusof the given imaging unit comprises a focusing optical element and atleast one second actuator associated therewith, wherein the step ofadjusting the optical focus comprises moving, via the at least onesecond actuator, the focusing optical element along an optical axis ofthe camera.
 13. The method of claim 9, wherein the optical element andthe means for adjusting the optical focus of the given imaging unit areimplemented together as a dynamically-controllable optical element,wherein the method further comprises dynamically changing the focallengths of the first optical portion and the second optical portion ofthe optical element.
 14. The method of claim 9, wherein the step ofadjusting the optical focus of the given imaging unit comprisesadjusting, based on the gaze direction of the user, at least onefocusing parameter of the optical element.
 15. The method of claim 9,wherein the first optical portion substantially surrounds the secondoptical portion, wherein a first focal length of the first opticalportion is smaller than a second focal length of the second opticalportion.
 16. The method of claim 9, wherein the optical element isrotationally asymmetric, the given imaging unit comprising at least onethird actuator associated with the optical element, wherein the methodfurther comprises controlling the at least one third actuator to adjusta rotational orientation of the optical element according to the gazedirection of the user.